[GLUTEN-11588][FLINK] Support rocksdb state for window operator#11589
Merged
PHILO-HE merged 1 commit intoapache:mainfrom Mar 6, 2026
Merged
[GLUTEN-11588][FLINK] Support rocksdb state for window operator#11589PHILO-HE merged 1 commit intoapache:mainfrom
PHILO-HE merged 1 commit intoapache:mainfrom
Conversation
lgbo-ustc
reviewed
Feb 10, 2026
gluten-flink/docs/Flink.md
Outdated
|
|
||
| **Get & compile RocksDB** | ||
| ```bash | ||
| wget https://github.com/ververica/frocksdb/archive/refs/heads/FRocksDB-6.20.3.zip |
KevinyhZou
changed the title
KevinyhZou
force-pushed
the
support_rocksdb_state
branch
from
5762e8c to
4a753a2
Compare
lgbo-ustc
reviewed
Feb 28, 2026
| inClass, | ||
| outClass, | ||
| sourceOperator.getDescription()); | ||
| if (sourceOperator instanceof WindowAggOperator) { |
Contributor
There was a problem hiding this comment.
should use if-else
if (sourceOperator instanceof WindowAggOperator) {
// create new windowAggOperator
offloadedOpConfig.setStreamOperator(newOneInputOp);
} else {
// the default handle
offloadedOpConfig.setStreamOperator(newOneInputOp);
}| inClass, | ||
| outClass, | ||
| sourceOperator.getDescription()); | ||
| if (sourceOperator instanceof WindowAggOperator) { |
Contributor
There was a problem hiding this comment.
Let's make the code simpler
- Add a new method in
GlutenOneInputOperator
public <NIN, NOUT> GlutenOneInputOperator<NIN, NOUT> cloneWithInputOutputClasses(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new GlutenOneInputOperator<>(
this.glutenPlan,
this.id,
this.inputType,
this.outputTypes,
newInClass,
newOutClass,
this.description);
}- Override
cloneWithInputOutputClassesinWindowAggOperator
@Override
public <NIN, NOUT> WindowAggOperator<NIN, NOUT, W> cloneWithInputOutputClasses(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new WindowAggOperator<>(
getPlanNode(),
getId(),
getInputType(),
getOutputTypes(),
newInClass,
newOutClass,
getDescription(),
keyType,
accNames,
accTypes);
}- Then only need to make following in
OffloadedJobGraphGenerator
GlutenOneInputOperator<?, ?> newOneInputOp = sourceOperator.cloneWithInputOutputClasses(inClass, outClass);
Contributor
Author
|
@PHILO-HE can you help to review this pr? thanks. |
PHILO-HE
changed the title
PHILO-HE
reviewed
Mar 5, 2026
Member
PHILO-HE
left a comment
PHILO-HE
left a comment
There was a problem hiding this comment.
Some minor comments. Thanks.
.github/workflows/flink.yml
Outdated
| sudo .github/workflows/util/install-flink-resources.sh | ||
| git clone -b gluten-0530 https://github.com/bigo-sg/velox4j.git | ||
| cd velox4j && git reset --hard 288d181a1b05c47f1f17339eb498dd6375f7aec8 | ||
| git clone -b fix_state_init https://github.com/KevinyhZou/velox4j.git |
Member
There was a problem hiding this comment.
Recommend to use organization repo. And please also make it consistent with the commit ID provided in Flink.md
| } | ||
|
|
||
| function install_rocksdb { | ||
| wget_and_untar https://github.com/ververica/frocksdb/archive/refs/heads/FRocksDB-${FROCKSDB_VERSION}.zip "" zip |
Member
There was a problem hiding this comment.
For downloading github repo code, maybe, better to add and use github_checkout function. Then, keep wget_and_untar unchanged.
KevinyhZou
force-pushed
the
support_rocksdb_state
branch
from
9843cfb to
3222fa8
Compare
KevinyhZou
force-pushed
the
support_rocksdb_state
branch
from
ac188da to
889b14f
Compare
Contributor
Author
|
Fixed. |
lgbo-ustc
approved these changes
Mar 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
support rocksdb state for window operator
this pr relies on: bigo-sg/velox4j#21 bigo-sg/velox#21
How was this patch tested?
manual tests
Was this patch authored or co-authored using generative AI tooling?
Related issue: #11588